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PREFACE 

The Agriculture and Resources Inventory Surveys Through Aerospace Remote Sens- 
ing is an 8-year program of research, development, evaluation, and application 
of aerospace remote sensing for agricultural resources, which began in fiscal 
year 1980. This program is a cooperative effort of the National Aeronautics 
and Space Administration, the U.S. Agency for International Development, and 
the U.S. Departments of Agriculture, Commerce, and the Interior. 

The work which is the subject of this document was performed within the Earth 
Resources Applications Division, Space and Life Sciences Directorate, at the 
Lyndon B. Johnson Space Center, National Aeronautics and Space Administration. 
Under Contract NAS 9-15800, personnel of Lockheed Engineering and Management 
Services Company, Inc., performed the tasks which contributed to the 
completion of this research. 
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EXECUTIVE SUMMARY 

In order to perform the 1981 U.S. Corn/Soybeans Pilot Experiment, a segment- 
level proportion estimation procedure and a procedure for aggregating segment- 
level proportion estimates to produce large-area acreage estimates were 
needed. 

The U.S. Corn/Soybeans Exploratory Experiment described in this report was 
performed to obtain a better understanding of the performance of the technol- 
ogy before proceeding to the pilot t ;eriment evaluation under a larger set of 
agricultural conditions. The experiment investigated the use of techniques 
for estimating corn and soybeans proportions which were similar to the tech- 
niques used for small grains proportion estimation. An evaluation was made of 
an aggregation procedure which was developed for further evaluation in the 
pilot experiment under different conditions. 

The results of the proportion estimation procedure evaluation indicated that, 
while the labeling part of the procedure worked quite well, the machine clas- 
sification part was not effective in improving the proportion estimates based 
on labeling information alone. The aggregation procedure performed quite well 
in producing large-area estimates of acreage and production. 

The evaluation indicated that the technology was capable of producing gener- 
ally acceptable area estimates for corn and soybeans. There were problems 
with certain aspects of the technology, but the problems were understood and 
could be addressed in on-going development activities. The development of the 
proportion estimation procedure for the pilot experiment relied to a great 
extent on the understanding of the technology resulting from the exploratory 
experiment. The evaluation of the aggregation procedure showed that it was 
rea<iy for use in the pilot experiment. 
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1. PURPOSE AND SCOPE 


The purpose of this report is to summarize the results from the U.S. Corn/ 
Soybeans Exploratory Experiment. This experiment was completed du^-ing fiscal 
year (FY) 1980. It was conducted as part of the Foreign Commodity Production 
Forecasting (FCPF) project of the Agriculture and Resources Inventory Surveys 
Through Aerospace Remote Sensing (AgRISTARS) program. 
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2. INTRODUCTION 


2.1 AqRI STARS PROGRAM 

The AgRISTARS program is an 8-year program of research, development, and evalu- 
ation of the application of aerospace remote sensing in monitoring agricultural 
resources. The program began in FY 1980. The AgRISTARS program is a coopera- 
tive effort of the National Aeronautics and Space Administration (NASA), the 
U.S. Agency for International Development (AID), and the U.S, Departments of 
Agriculture, Commerce, and the Interior (USDA, USDC, and USDI). The goal of 
this program is to determine the usefulness, cost, and extent to which aero- 
space remote sensing data can be used by the USDA to improve the objectivity, 
reliability, and timeliness of information required to carry out USDA missions 
(ref. 1). 

2.2 FCPF PROJECT 

An important component of the AgRISTARS program is the FCPF project. The 
objective of the FCPF project is to develop and test procedures for using aero- 
space remote sensing technology to provide more objective, timely, and reliable 
crop production forecasting in foreign areas. To develop technology for use in 
foreign areas, the FCPF project builds upon existing remote sensing technology 
and extends this technology to additional crops and regions (ref. 2). 

During the first year of the FCPF project, two exploratory experiments were 
performed using U.S. data to develop and evaluate techniques. These experi- 
ments were the U.S ./Canada Wheat and Barley Exploratory Experiment (1) and the 
U.S. Corn/Soybean Exploratory Experiment. This report presents the results 
from the U.S. Corn/Soybean Exploratory Experiment. 

2.3 U.S. CORN/SOYBEAN EXPLORATORY EXPERIMENT 

The overall objective for the U.S. Corn/Soybean Exploratory Experiment was to 
evaluate segment-level labeling and proportion estimation techniques and to 
test a procedure for making aggregated estimates of area and production. These 
two components plus a yield estimation component form the nucleus of a system 
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for nk\k1ng large area crop acreage and production estimates. The results of 
these evaluations provide the basis for establishing baseline procedures for 
subsequent experiments and for adaptation of the procedures for use in foreign 
regions. 

The L'.S. Corn/Soybean Exploratory Experiment was the first attempt" to adapt the 
segment-level proportion estimation techniques developed for small grains to 
other crops. The segment-level proportion estimates were obtained by labeling 
selected pixels from the segment as training for a maximun likelihood classi- 
fier. The results from the classification were corrected for bias using an 
independent set of labeled pixels. The pixel labeling was done using an 
objective procedure based on labeling techniques developed Airing previous 
experiments. This marks the first time an objective procedure was used to 
label pixels instead of relying on the experience and insight of highly trained 
analysts to obtain pixel labels. 

In order to test the weighted aggregation procedure proposed for the FCPF 
project, a simulation test was performed as part of this exploratory experiment. 
The weighted aggregation procedure was set up to make the best use of historical 
data to stabilize large area aggregated estimates in regions where there is a 
high rate of data loss or where ttiere are large classification errors. 
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3. EXPERIMENT DESCRIPTION 


Two tests were performed as part of this exploratory experiment. The first 
test was the Classification Procedures Verification Test (CPVT). The second 
test was the Simulated Aggregation Test (SAT). 

3.1 CLASSIFICATION PROCEDURES VERIFICATION TEST DESCRIPTION 

The CPVT consisted of labeling and proportion estimation on 25 segments from 
four agrophysical units (APU's) In the U.S. Corn Belt using Landsat data from 
the 1978 crop year. The locations of the segments used In the CPVT are shown 
In figure 1, 

The two objectives of this test were (1) to determine the accuracy of the newly 
developed objective labeling procedure and recommend Improvements for use In 
the SAT and (2) to determine the accuracy of the proportion estimation 
procedure. 

The procedure used to process the segments for this test Is shown In figure 2. 
Using Landsat and ancillary data, an objective labeling procedure (ref. 3) was 
used to label two sets of p1),els from each se^nt. The major steps In the 
labeling procedure are shown In figure 3. The procedure Is set up to provide 
Increasingly detailed labeling Information at each step 1n the procedure. The 
first step consists of a decision tree labeling logic which Is used to separate 
the pixels Into cropland and iioncropland. The pixels labeled cropland In the 
first step are separated Into summer crops and "other crops" In the second 
step. This step also uses a decision tree labeling logic. In the third step, 
a greenness/brightness scatter plot for the separation acquisition Is used for 
separating the summer crop pixels Into corn and soybeans. 

The first set of analyst-labeled pixels (Type I dots) Is used as training for a 
clustering algorithm which grouped all of the pixels In the segment into 
clusters on the basis of their spectral values. Each of the resulting clusters 
Is labeled as corn, soybeans, or "other" using the labeled Type I dot closest 
to the mecin of the cluster. On the basis of the means and variances for each 
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U.S. CORN BELT PROCEDURES VERIFICATION TEST 



Figure 1.- Map showing locations of the segments used in the 
Classification Procedures Verification Test. 
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Figure 2.- Diagram showing procedure for processing segment for 
the Classification Procedures Verification test. 
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Figure 3,- Diagram showing the 
for the Classification 


major steps in the labeling procedure 
Procedures Verification test. 
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cluster, a maximum likelihood classification of every pixel in the segment is 
performed. Using the second set of analyst-labeled dots (Type 2 dots) as a 
random sample of the segment, the proportion based on the classification is 
corrected for any bias introduced by the classification process. 

The segments in the CPVT were processed independently by three groups of 
analysts. Each segment was processed by at least two of the groups. The test 
followed a rigid experiment design so that analysis of variance techniques 
could be used to determine if the quality of the labeling and proportion esti- 
mation results were dependent on the group doing the labeling or on the APU in 
which the segment was located (ref. 4). All of the evaluations were performed 
by comparing the labeling and classification results to the digitized 
ground-truth crop inventories. 

An error characterization study was performed to determine if any changes could 
be made in the labeling procedure to improve the accuracy of the labeling. 

These changes were incorporated into the procedure before it was used in the 
second test. 

An evaluation was made to determine the effectiveness of the classification 
procedure in producing proportion estimates. 

3.2 SIMULATED AGGREGATION TEST DESCRIPTION 

The second test performed as part of this exploratory experiment was the SAT. 

It involved labeling and proportion estimation for 88 segments in the Corn Belt 
using Landsat data from the 1978 crop year. These results were used to estimate 
the variability of proportion estimates for a simulation test of the aggregation 
procedure. There were two parts to the test. The first part was the actual 
labeling and classification for the segments. The objective of this part was to 
evaluate the labeling and proportion estimation procedures as they were modified 
following the CPVT and to provide estimates of the variability introduced by the 
labeling and proportion estimation procedures for the aggregation procedure 
simulation test. The second part of the test was the actual aggregation pro- 
cedure simulation itself. The objectives of the simulation were to verify that 
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the sample allocation procedure provided correct sample allocations among the 
strata, to validate the aggregation and variance estimation logic, and to 
determine the effect of random nonresponse on aggregated estimates produced by 
the procedure. 

The labeling procedure used In the SAT was essentially the same as that used In 
the CPVT. The changes made as a result of the CPVT were mainly Improvements In 
the clarity of the procedure. The proportion estimation procedure was modified 
from the procedure used In the CPVT. Based on a study performed by Supporting 
Research (ref. 5) and on resource considerations, the decision was made not to 
perform the bias correction on the Initial proportion estimates In the SAT. 
Therefore, the proportion estimation procedure Involved labeling of the Type I 
dots, classification of the segment, and proportion estimation by enumeration 
of pixels In the class of Interest. 

The aggregation procedure tested In the second part of the SAT consisted of a 
technique for using historical data to compensate for the loss of data In a 
particular stratum (ref. 6). The technique Involves a weighting procedure 
which places more reliance on historical data as the classification results 
become less reliable because of data loss or errors In the classification 
results. 

The 88 segmei^ts In the SAT were each processed once. Twenty- three of the seg- 
ments had been processed In the CPVT. These were processed In the SAT, but by 
a different analys* ^roup. Thirty-five additional segments with ground-truth 
Inventories were ,'ocessed and used In the evaluations. The locations of the 
segments used In this test are shown In figure 4. Evaluation of the labeling 
and proportion estimation accuracies were performed using the segments for 
which ground-truth Information was available,, 

The simulation test of the aggregation procedure was performed by setting up an 
allocation of 204 simulated segments In 12 strata In the states of Illinois, 
Indiana, and Iowa. Historical data were used to determine the mean crop pro- 
portions for each stratum. The distribution of segment-level crop proportions 
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SIMULATED AGGREGATION TEST 
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Figure 4. 


Map showing locations of the segments used in the 
Simulated Aggregation Test. 


within strata was determined from the historical variability and from the 
empirical variances observed In the classification results. State-level his- 
torical data were used to determine mean yields, and the distribution of yield 
estimates was determined using ESS yield-model variance. A Monte Carlo simula- 
tion was performed in which segments were randomly designated as "lost". For 
each loss rate, 100 slirwlatlons were performed to obtain aggregated estimates 
of production. 
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4. EXPERIMENT RESULTS 


4.1 CLASSIFICATION PROCEDURES VERIFICATION TEST RESULTS 

In the CPVT, statistical tests were performed to determine if there was a sig- 
nificant difference in the quality of the labeling and proportion estimation 
results due to the group performing the processing or the region in which the 
segment was located (ref. 7). The measures of quality used were dot labeling 
accuracy, percentage of correct classification, and proportion estimation 
error. A regional difference was observed for the dot labeling accuracy for 
soybeans. The labeling of soybeans was significantly worse in a predominantly 
corn-producing region than in the regions where soybeans were more prevalent. 

A group effect was found in the dot labeling accuracy for corn. One group pro- 
duced significantly less accurate dot labeling for corn. Investigation showed 
that the difference was due to a difference in the way the group placed the 
separation line on the scatter plots for corn and soybeans. 

The labeling accuracies for the CPVT are shown in table 1. The labeling 
accuracy is comparable to the small grains labeling accuracies previously 
achieved during the LACIE. The labeling for Type I dots was better than for 
Type II dots. This difference results from the fact that the Type I dots are 
required to be spectrally pure, while the Type II dots can be spectrally mixed. 
It is, therefore, natural to expect better labeling accuracy on dots which are 
representative of a particular crop, rather than a mixture of signatures from 
more than one crop. 

The proportion estimation errors as a function of the true proportion are shown 
for both corn and soybeans in figure 5. The average proportion of corn in the 
segments was 38 percent. The machine processing procedure underestimated the 
corn proportion by an average of 4 percent. The average proportion of soybeans 
was 28 percent. The procedure underestimated the soybeans proportion by 6 per- 
cent. All of the bias and half of the variability in the proportion estimation 
errors were the result of dot labeling errors. The proportion estimates pro- 
duced by the procedure were not any better than estimates obtained by using the 
Type II dots as a random sample. Therefore, the machine processing (i.e., 
clustering and classification) did not improve the results. 
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TABLE 1.- SUMMARY OF DOT LABELING RESULTS FOR THE 
CLASSIFICATION PROCEDURES VERIFICATION TEST 


Ground- truth 
category 

Percent correctly labeled 

Type 1 dots 

Type 2 dots 

Corn 

83 

73 

Soybeans 

79 

64 

Other 

93 

86 

All categories 

86 

75 
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Proportion estimation error 




20 40 60 80 
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Mean error = 

-4% 

Mean error = 

-6% 

Standard deviation = 

8% 

Standard deviation = 

1 % 

RMSE 

9% 

RMSE 

9% 


(a) Corn — bias-corrected 
estimate. 


(b) Soybeans — bias-corrected 
estimate. 


Figure 5.- Proportion estimation errors as a function of the 
true proportion for both corn and soybeans. 
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Since the labeling and classification accuracies were much better for spectrally 
pure pixels than for mixed pixels, a study was maoe on the segments in this test 
to determine if accurate proportion estimates could be obtained from classifica- 
tion information for spectrally pure pixels. In order to perform the study, 
each of the pure pixels was given its ground-truth label, and a proportion esti- 
mate was made using only these pixels. Figure 6 shows the proportion estimation 
errors for two criteria for pixel purity. Pixels which meet the “one-half 
pixel" purity criterion are at least one-half pixel from the field boundaries. 
Pixels which meet the "one pixel" criterion are at least one pixel from the 
field boundaries. The results indicate that proportion estimates based only on 
pure pixels can be biased and have a great de^l of variability. In the data set 
used in this test, the corn estimates showed a positive bias. 

4.2 SIMULATED AGGREGATION TEST RESULTS 

In the SAT, the labeling accuracy improved over the accuracy in the CF" 

(ref. 8). Table 2 shows a comparison of the labeling accuracies in the two 
tests. The improvement in the labeling accuracy for the second test was due to 
changes in the labeling procedure recommended on the basis of the first test 
and to an improved procedure for selecting acquisitions. 

The proportion estimation results for the SAT are shown in figure 7. The 
results for soybeans proportion estimation were comparable to those obtained in 
the CPVT. The average soybeans proportion in the segments was 30 percent. The 
procedure underestimated the soybeans proportions by an average of 8 percent. 

For corn, the average proportion was 41 percent. In the SAT, the procedure 
overestimated the corn proportions by 5 percent, while in the CPVT, the propor- 
tions were underestimated by 4 percent. The change in bias between the two 
tests is due to the fact that a bias correction was not performed in the SAT. 

The proportions based on the pixel -level classification results are too high 
because the classification procedure was trained using only spectrally pure 
pixels. Training with only pure pixels produces a classification which is 
representative of the pure areas of the segment, rather than of the entire 
segment. As the pure pixel studies showed, this will produce a positive bias 
in the classification results. 
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Pure pixel proportion error 
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Mean error * 11% 
Standard deviation * 9% 
RMSE * 14% 



Mean error * 2% 
Standard deviation * 6% 
RMSE » 6% 


pixels are used to determine 



TABLE i,- COMPARISON OF LABELING ACCURACY FOR 
CPVT AND SAT TESTS 


Ground- truth 

1 Percent correctly labeled 

categories 

CPVT 

(Type I dots) 

SAT 

Corn 

86 

93 

Soybeans 

79 

88 

Other 

93 

96 

All categories 

86 

92 
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Figure 7.- Summary of results for the simulated Aggregation Test. 
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The slnwlatlon tests of the sampling and aggregation procedures (ref. 9) were 
set up to provide large area production estimates with a coefficient of vari- 
ation (CV) of 5 percent for both corn and soybeans at an acquisition rate of 
100 percent. The aggregation procedure was tested to determine If the CV 
estimates confuted by the procedure were correct. If nonresponse Introduced any 
bias Into the aggregated estimates, and If the CV's at reduced response rates 
were re^Lonable. 

The simulation tests showed that the allocation proceAjre was producing 
estimates with CV's In good agreement with the expected v6lue of 5 percent 
(CV = 4.7% for corn and CV = 5.2% for soybeans). The tests of the weighted 
aggregation procedure demonstrated that the procedure Introduced no bias Into 
the aggregated area and production estimates for acquisition rates as low as 
10 percent. Figure 8 shows the CV's resulting from reduced acquisition rates 
for area and for production. These variances are reasonable, and the average 
CV estimates produced by the procedure correspo id closely to the CV's of the 
simulated sample. 
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production for three states. 




5. CONCLUSIONS AND RECOMMENDATIONS 


The results from the labeling evaluations indicate that the corn/soybeans 
labeling procedure performs very well in the U.S, Corn Belt with full -season 
(after tassel ing) Landsat data. The procedure should be readily adaptable to 
corn/soybeans labeling required for subsequent exploratory experiments or pilot 
tests. 

The machine classification procedures evaluated in this experiment were not 
effective in improving the proportion estimates. The corn proportions produced 
by the machine procedures had a large bias when the bias correction was not 
performed. This bias v;as caused by the manner in which the machine procedures 
handled spectrally impure pixels. Alternatives to the machine processing tech- 
niques used in this experiment should be investigated to see if more effective 
techniques can be found. 

The sinwlation test indicated that the weighted aggregation procedure performed 
quite well. Although further work can be done to in^rove both ths simulation 
tests and the aggregation procedure, the results of this test show that the 
procedure should serve as a useful baseline procedure in future exploratory 
experiments and pilot tests. 
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